Iterative image reconstruction

ABSTRACT

Systems and methods are disclosed for performing operations comprising: accessing a current structural estimate of a region of interest; generating a first simulated X-ray measurement based on the current structural estimate of the region of interest; receiving a first real X-ray measurement; and generating an update to the current structural estimate of the region of interest as a function of the first simulated X-ray measurement and the first real X-ray measurement, the update being generated invariant on the current structural estimate.

TECHNICAL FIELD

Embodiments of the present disclosure pertain generally to iterativeimage reconstruction.

BACKGROUND

Radiation therapy (or “radiotherapy”) can be used to treat cancers orother ailments in mammalian (e.g., human and animal) tissue. One suchradiotherapy technique involves irradiation with a Gamma Knife, wherebya patient is irradiated by a large number of low-intensity gamma raybeams that converge with high intensity and high precision at a target(e.g., a tumor). In another embodiment, radiotherapy is provided using alinear accelerator, whereby a tumor is irradiated by high-energyparticles (e.g., electrons, protons, ions, high-energy photons, and thelike). The placement and dose of the radiation beam must be accuratelycontrolled to ensure the tumor receives the prescribed radiation, andthe placement of the beam should be such as to minimize damage to thesurrounding healthy tissue, often called the organ(s) at risk (OARs).Radiation is termed “prescribed” because a physician orders a predefinedamount of radiation to the tumor and surrounding organs similar to aprescription for medicine. Generally, ionizing radiation in the form ofa collimated beam is directed from an external radiation source toward apatient.

A specified or selectable beam energy can be used, such as fordelivering a diagnostic energy level range or a therapeutic energy levelrange. Modulation of a radiation beam can be provided by one or moreattenuators or collimators (e.g., a multi-leaf collimator (MLC)). Theintensity and shape of the radiation beam can be adjusted by collimationto avoid damaging healthy tissue (e.g., OARs) adjacent to the targetedtissue by conforming the projected beam to a profile of the targetedtissue.

The treatment planning procedure may include using a three-dimensional(3D) image of the patient to identify a target region (e.g., the tumor)and to identify critical organs near the tumor. Creation of a treatmentplan can be a time-consuming process where a planner tries to complywith various treatment objectives or constraints (e.g., dose volumehistogram (DVH), overlap volume histogram (OVH)), taking into accounttheir individual importance (e.g., weighting) in order to produce atreatment plan that is clinically acceptable. This task can be atime-consuming trial-and-error process that is complicated by thevarious OARs because as the number of OARs increases (e.g., up tothirteen for a head-and-neck treatment), so does the complexity of theprocess. OARs distant from a tumor may be easily spared from radiation,while OARs close to or overlapping a target tumor may be difficult tospare.

Traditionally, for each patient, the initial treatment plan can begenerated in an “offline” manner. The treatment plan can be developedwell before radiation therapy is delivered, such as using one or moremedical imaging techniques. Imaging information can include, forexample, images from X-rays, computed tomography (CT), nuclear magneticresonance (MR), positron emission tomography (PET), single-photonemission computed tomography (SPECT), or ultrasound. A health careprovider, such as a physician, may use 3D imaging information indicativeof the patient anatomy to identify one or more target tumors along withthe OARs near the tumor(s). The health care provider can delineate thetarget tumor that is to receive a prescribed radiation dose using amanual technique, and the health care provider can similarly delineatenearby tissue, such as organs, at risk of damage from the radiationtreatment. Alternatively, or additionally, an automated tool (e.g., ABASprovided by Elekta AB, Sweden) can be used to assist in identifying ordelineating the target tumor and organs at risk. A radiation therapytreatment plan (“treatment plan”) can then be created using anoptimization technique based on clinical and dosimetric objectives andconstraints (e.g., the maximum, minimum, and fraction of dose ofradiation to a fraction of the tumor volume (“95% of target shallreceive no less than 100% of prescribed dose”), and like measures forthe critical organs). The optimized plan is comprised of numericalparameters that specify the direction, cross-sectional shape, andintensity of each radiation beam.

The treatment plan can then be later executed by positioning the patientin the treatment machine and delivering the prescribed radiation therapydirected by the optimized plan parameters. The radiation therapytreatment plan can include dose “fractioning,” whereby a sequence ofradiation treatments is provided over a predetermined period of time(e.g., 30-45 daily fractions), with each treatment including a specifiedfraction of a total prescribed dose. However, during treatment, theposition of the patient and the position of the target tumor in relationto the treatment machine (e.g., linear accelerator—“linac”) is veryimportant in order to ensure the target tumor and not healthy tissue isirradiated.

Since most patients receive more than one fraction of radiation as partof a course of therapy, and because the anatomy may change (deform)between these fractions, it is not straightforward to sum the dosesdelivered during the individual fractions so the physician canaccurately gauge how the treatment is proceeding relative to theoriginal intent as defined by the prescription.

Overview

In some embodiments, a system is provided that includes: a memory; andone or more processors that, when executing instructions stored in thememory, are configured to perform operations comprising: accessing acurrent structural estimate of a region of interest; generating a firstsimulated X-ray measurement based on the current structural estimate ofthe region of interest; receiving a first real X-ray measurement; andgenerating an update to the current structural estimate of the region ofinterest as a function of the first simulated X-ray measurement and thefirst real X-ray measurement, the update being generated invariant onthe current structural estimate.

In some implementations, the first real X-ray measurement is receivedfrom a cone-beam computed tomography (CBCT) system or computedtomography (CT) system, and wherein the operations further comprisecomputing the update as a derivative of a statistical objective functionwith respect to the first simulated X-ray measurement.

In some implementations, the operations further comprise: linearlyprojecting the update to the current structural estimate into an imagespace to form a perturbation; scaling the perturbation by a scalar forstability; and subtracting the scaled perturbation from the currentstructural estimate to generate an updated structural estimate.

In some implementations, the operations further comprise: applying atleast one of regularization, momentum or denoising to the updatedstructural estimate.

In some implementations, the current structural estimate comprises anX-ray attenuation map that represents a three-dimensional (3D) model ofthe region of interest.

In some implementations, the X-ray attenuation map comprises a linearX-ray attenuation map.

In some implementations, generating the first simulated X-raymeasurement based on the current structural estimate of the region ofinterest comprises applying the current structural estimate to a modelthat generates an expected output of a real X-ray measurement.

In some implementations, the current structural estimate comprises anX-ray attenuation map, and wherein the model generates the expectedoutput of the real X-ray measurement, for a given measurement index iper number of detector elements in a cone-beam computed tomography(CBCT) system, in accordance with: z_(i)=b_(i)*exp(−[Ax]_(i))+r_(i),where b is an incident intensity of flood field, A is a systemprojection matrix describing a combination of each image pixel at eachdetector element, x is the current X-ray attenuation map, exp is anexponential function, and r is background noise including scatter.

In some implementations, the operations further comprise re-estimating rrepresenting the background noise including scatter based on the currentstructural estimate at each iteration of generating the update.

In some implementations, the model comprises a modeling functionrepresenting beam hardening from a polyenergetic source, the modelingfunction comprising a machine learning technique that is trained toestablish a relationship between a training real X-ray measurement and atraining known simulated X-ray measurement; a linear model; or anon-linear model that fits X-ray data to some nominal value comprisingat least one of relative electron density, mass density, monoenergeticattenuation, proton stopping power, or bone mineral density.

In some implementations, the operations further comprise repeating thegenerating of a simulated X-ray measurement, receiving of a real X-raymeasurement, and generating of an update to the current structuralestimate for multiple sets of simulated and real X-ray measurements.

In some implementations, the operations further comprise: accessing anupdated structural estimate of the region of interest; generating asecond simulated X-ray measurement based on the updated structuralestimate of the region of interest; receiving a second real X-raymeasurement; and generating a further update to the updated structuralestimate of the region of interest as a function of the second simulatedX-ray measurement and the second real X-ray measurement.

In some implementations, the operations further comprise: accessing anobjective function comprising a negative log-likelihood (NLL) function;computing a loss by applying the NLL function to a combination of thefirst simulated X-ray measurement and the first real X-ray measurement;and in response to determining that the loss fails to satisfy acriterion, performing the update to the current structural estimate, thecriterion comprising a difference between adjacent updates falling belowa threshold, a number of iterations falling below a maximum iterationvalue, an elapsed time falling below a maximum time limit, or inputrequesting termination and display of a result.

In some implementations, the current structural estimate comprises anX-ray attenuation map, and wherein the update to the current structuralestimate of the region of interest is computed in accordance with:A^(T)(y/(b*exp(−Ax)+r)−1), where A is a system projection matrixdescribing a combination of each image pixel at each detector element,A^(T) is a transpose of the system projection matrix, y is the firstreal X-ray measurement, b is an incident intensity of flood field, x isthe current X-ray attenuation map, exp is an exponential function, and ris background noise including scatter.

In some implementations, the operations further comprise: receiving aplurality of real X-ray measurements; partitioning the plurality of realX-ray measurements into measurement groups, a first measurement group ofthe measurement groups corresponding to a group of X-ray measurementsused to update the current structural estimate, and a second measurementgroup of the measurement groups corresponding to a group of X-raymeasurements that is skipped from updating the current structuralestimate; determining that the first real-X-ray measurement falls withinthe first measurement group; and in response to determining that thefirst real-X-ray measurement falls within the first measurement group,performing the update to the current structural estimate.

In some implementations, the operations further comprise: collecting aplurality of updates to the structural estimate, each of the pluralityof updates being associated with a respective iteration; identifying apattern of updates based on the collected plurality of updates; andperforming the update to the current structural estimate based on theidentified pattern of updates.

In some embodiments, a method is provided for accessing a currentstructural estimate of a region of interest; generating a firstsimulated X-ray measurement based on the current structural estimate ofthe region of interest; receiving a first real X-ray measurement; andgenerating an update to the current structural estimate of the region ofinterest as a function of the first simulated X-ray measurement and thefirst real X-ray measurement, the update being generated invariant onthe current structural estimate.

In some implementations, the method includes computing the update as aderivative of a statistical objective function with respect to the firstsimulated X-ray measurement.

In some implementations, the method includes linearly projecting theupdate to the current structural estimate into an image space to form aperturbation; scaling the perturbation by a scalar for stability; andsubtracting the scaled perturbation from the current structural estimateto generate an updated structural estimate.

In some implementations, the method includes applying at least one ofregularization, momentum or denoising to the updated structuralestimate.

In some implementations, the current structural estimate comprises anX-ray attenuation map that represents a three-dimensional (3D) model ofthe region of interest.

In some implementations, the X-ray attenuation map comprises a linearX-ray attenuation map.

In some implementations, generating the first simulated X-raymeasurement based on the current structural estimate of the region ofinterest comprises applying the current structural estimate to a modelthat generates an expected output of a real X-ray measurement.

In some embodiments, a non-transitory computer-readable medium isprovided that includes non-transitory computer-readable instructionsthat, when executed by one or more processors, configure the one or moreprocessors to perform operations comprising: accessing a currentstructural estimate of a region of interest; generating a firstsimulated X-ray measurement based on the current structural estimate ofthe region of interest; receiving a first real X-ray measurement; andgenerating an update to the current structural estimate of the region ofinterest as a function of the first simulated X-ray measurement and thefirst real X-ray measurement, the update being generated invariant onthe current structural estimate.

In some implementations, the operations further comprise: computing theupdate as a derivative of a statistical objective function with respectto the first simulated X-ray measurement.

In some implementations, the operations further comprise: linearlyprojecting the update to the current structural estimate into an imagespace to form a perturbation; scaling the perturbation by a scalar forstability; and subtracting the scaled perturbation from the currentstructural estimate to generate an updated structural estimate.

In some implementations, the operations further comprise: applying atleast one of regularization, momentum or denoising to the updatedstructural estimate.

In some implementations, the current structural estimate comprises anX-ray attenuation map that represents a three-dimensional (3D) model ofthe region of interest.

In some implementations, the X-ray attenuation map comprises a linearX-ray attenuation map.

In some implementations, generating the first simulated X-raymeasurement based on the current structural estimate of the region ofinterest comprises applying the current structural estimate to a modelthat generates an expected output of a real X-ray measurement.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsdescribe substantially similar components throughout the several views.Like numerals having different letter suffixes represent differentinstances of substantially similar components. The drawings illustrategenerally, by way of example but not by way of limitation, variousembodiments discussed in the present document.

FIG. 1 illustrates an example radiotherapy system, according to someembodiments of the present disclosure.

FIG. 2A illustrates an example radiation therapy system that can includeradiation therapy output configured to provide a therapy beam, accordingto some embodiments of the present disclosure.

FIG. 2B illustrates an example system including a combined radiationtherapy system and an imaging system, such as a cone beam computedtomography (CBCT) imaging system, according to some embodiments of thepresent disclosure.

FIG. 3 illustrates a partially cut-away view of an example systemincluding a combined radiation therapy system and an imaging system,such as a nuclear MR imaging (MRI) system, according to some embodimentsof the present disclosure.

FIGS. 4A and 4B depict the differences between an example MRI image anda corresponding CT image, respectively, according to some embodiments ofthe present disclosure.

FIG. 5 illustrates an example collimator configuration for shaping,directing, or modulating an intensity of a radiation therapy beam,according to some embodiments of the present disclosure.

FIG. 6 illustrates an example Gamma Knife radiation therapy system,according to some embodiments of the present disclosure.

FIG. 7 illustrates an example flow diagram for deep learning, accordingto some embodiments of the present disclosure.

FIG. 8 illustrates an example data flow for training and use of amachine learning model to generate a simulated X-ray measurement orupdate to a structural estimate, according to some embodiments of thepresent disclosure.

FIG. 9 illustrates a method for generating an update to a structuralestimate of a region of interest, according to some embodiments of thepresent disclosure.

FIG. 10 depicts the differences between structural estimates generatedaccording to different techniques, according to some embodiments of thepresent disclosure.

FIG. 11 illustrates an example block diagram of a machine on which oneor more of the methods as discussed herein can be implemented.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and which is shown byway of illustration-specific embodiments in which the present disclosuremay be practiced. These embodiments, which are also referred to hereinas “examples,” are described in sufficient detail to enable thoseskilled in the art to practice the disclosure, and it is to beunderstood that the embodiments may be combined, or that otherembodiments may be utilized, and that structural, logical and electricalchanges may be made without departing from the scope of the presentdisclosure. The following detailed description is, therefore, not betaken in a limiting sense, and the scope of the present disclosure isdefined by the appended claims and their equivalents.

Physically quantifying reconstructions from CT is highly valuable insome medical practices. In radiation therapy for example, the electrondensity inferred from CT images of the patient allows the dosedeposition from the treatment beam to be accurately modelled.Additionally, quantifying the bone mineral density allows osteoporosisto be characterized and the risk of bone fractures to be assessed. Dueto the non-linear energy-dependence of X-ray attenuation, however,mapping from a set of raw measurements to a consistent physicallyquantitative reconstruction (e.g., a structural estimate) is notstraightforward, and requires both actively accounting for thepolyenergetic effects such as beam hardening and establishing a relationto the quantity of interest.

Mapping from CT to physical density is usually treated in a several stepprocess: linearizing the measurements by correcting for scatter, takingthe logarithm and non-linearly calibrating from a polyenergetic toapproximate monoenergetic source; inverting the linearized projectionswith analytic or iterative reconstruction algorithms; then finallyapplying anon-linear calibration to mass or electron density. Oneapproach to mapping from a single polyenergetic to a monoenergeticmeasurement is to model the X-ray attenuation explicitly in terms ofphysical processes, given quantitative physical parameters of interest.One such choice is to model the photoelectric and Compton effects interms of relative atomic number and electron density, which isreasonably accurate for relevant elements and energies. Givenmeasurements from two sufficiently different spectra-a technique knownas dual-energy CT (DECT)-a projection of Compton attenuation can beuniquely determined, of which electron density is an analytic function.This DECT technique effectively bypasses the need for assumptions suchas water-bone compositions, so can be applicable to a wider range inmaterials.

X-ray scatter is a large source of errors in CT, where scattered X-rayscorrupt the line of sight attenuation models used during imagereconstruction (e.g., generation of the structural estimate). In CBCT,due to its large field of view, the magnitude of these interferingX-rays is commonly of the same order of magnitude as the signal ofinterest and can even be considerably higher. With this, artifacts andinaccuracies in the reconstruction are inevitable unless it can becorrectly compensated. A fundamental reason why estimating scatter isdifficult is that unlike modelling attenuation, requiring a single pathfrom source to detector for each measurement, one may need to considerevery possible path a photon can take through various numbers ofscattering events to get the full picture to generate the structuralestimate. This can therefore not only be dependent on the attenuatingmaterials and intensity along a single pencil beam, but it can bedependent on the full projection fluence and the complete structure ofthe specimen. Not only is calculating all these paths computationallyexhaustive, but one does not even have knowledge of the structure of thespecimen prior to reconstruction, since this is the task of thereconstruction itself. This therefore presents a dependency problem,where one may need the image to estimate scatter, but cannot form theimage accurately because of scatter contamination.

Some existing approaches calculate the perturbation (changes/updates) tothe structural estimate at each iteration as the gradient of astatistical objective function with respect to the structural estimateitself, which is also computationally exhaustive and still leads toinaccuracies in the structural estimate. Namely, existing approachesprovide iterative reconstruction methods that minimize an objectivefunction at each iteration, which is composed of data consistency andstructural penalty terms. These approaches, though, suffer from a largecomputational expense of each iteration, especially with high-resolutionmulti-dimensional data; poor convergence rate requiring many iterations;unstable and divergent behavior due to numerical approximations ornon-convexity; and the introduction of phantom artefacts from aninaccurate noise model, geometrical errors, inaccurate modelling of thephysics, or numerical errors.

The disclosed embodiments address these challenges by updating astructural estimate of a CBCT system (e.g., a real X-ray measurement)based on simulated measurements of the CBCT system which are invariantto (independent of) the structural estimate itself. This yields toincreased stability and a faster convergence rate and results in ahigher quality image in fewer iterations. In this way, the overallefficiency of the computing device is improved, as less resources areconsumed in generating a structural estimate of a CBCT system withgreater accuracy than traditional approaches. Specifically, CBCT imagesmay be generated using a kV imaging detector affixed to a linearaccelerator. These images can be used to generate 3D images for imageguidance and online adaptive radiotherapy prior to each radiationtreatment. Although very useful, these images have suffered from lowerimage quality than diagnostic CT in previous approaches. Iterative imagereconstruction can be used to improve the CBCT images and the disclosedtechniques improve both image quality and reconstruction time forgenerating such CBCT images over typical systems.

Specifically, according to some examples, a current structural estimate(e.g., an X-ray attenuation map) of a region of interest is accessed.The disclosed techniques generate a first simulated X-ray measurementbased on the current structural estimate of the region of interest andreceive a first real X-ray measurement from a CBCT system or CT system.The disclosed techniques generate an update to the current structuralestimate of the region of interest as a function of the first simulatedX-ray measurement and the first real X-ray measurement, in which theupdate is generated invariant on the current structural estimate. Thisresults in faster image reconstruction time for the structure estimate,which also has greater accuracy (e.g., undesirable phantom artefacts,including streaking and shading in the image, are reduced).

FIG. 1 illustrates an example radiotherapy system 100 for providingradiation therapy to a patient. The radiotherapy system 100 includes animage processing device 112. The image processing device 112 may beconnected to a network 120. The network 120 may be connected to theInternet 122. The network 120 can connect the image processing device112 with one or more of a database 124, a hospital database 126, anoncology information system (OIS) 128, a radiation therapy device 130,an image acquisition device 132, a display device 134, and a userinterface 136. The image processing device 112 can be configured togenerate radiation therapy treatment plans 142 to be used by theradiation therapy device 130.

The image processing device 112 may include a memory device 116, animage processor 114, and a communication interface 118. The memorydevice 116 may store computer-executable instructions, such as anoperating system 143, radiation therapy treatment plans 142 (e.g.,original treatment plans, adapted treatment plans and the like),software programs 144 (e.g., artificial intelligence, deep learning,neural networks, radiotherapy treatment plan software), and any othercomputer-executable instructions to be executed by the image processor114.

In one embodiment, the software programs 144 may convert medical imagesof one format (e.g., MRI) to another format (e.g., CT) by producingsynthetic images, such as pseudo-CT images. For instance, the softwareprograms 144 may include image processing programs to train a predictivemodel for convening a medical image 146 in one modality (e.g., an MRIimage) into a synthetic image of a different modality (e.g., a pseudo CTimage); alternatively, the trained predictive model may convert a CTimage into an MRI image. In another embodiment, the software programs144 may register the patient image (e.g., a CT image or an MR image)with that patient's dose distribution (also represented as an image) sothat corresponding image voxels and dose voxels are associatedappropriately by the network. In yet another embodiment, the softwareprograms 144 may substitute functions of the patient images or processedversions of the images that emphasize some aspect of the imageinformation. Such functions might emphasize edges or differences invoxel textures, or any other structural aspect useful to neural networklearning. In another embodiment, the software programs 144 maysubstitute functions of the dose distribution that emphasize some aspectof the dose information. Such functions might emphasize steep gradientsaround the target or any other structural aspect useful to neuralnetwork learning. The memory device 116 may store data, includingmedical images 146, patient data 145, and other data required to createand implement a radiation therapy treatment plan 142.

In yet another embodiment, the software programs 144 may generate astructural estimate (e.g., a 3D model of the region of interest) usingan iterative image reconstruction process. The structural estimate maybe or include an X-ray attenuation map that represents a 3D model of aregion of interest. The structural estimate may be used to estimate orsimulate X-ray measurements to be compared with real X-ray measurementsfor updating the structural estimate. Specifically, the softwareprograms 144 can access a current structural estimate of the region ofinterest and generate a first simulated X-ray measurement based on thecurrent structural estimate of the region of interest. A simulated X-raymeasurement, as referred to herein, represents the expected output of anX-ray detector element when an X-ray source projects one or more X-raybeams through the region of interest towards the X-ray detector element.The simulated X-ray measurement can provide an expected image outputthat is to be received from the X-ray detector element. The softwareprograms 144 can receive a first real X-ray measurement from a CBCTsystem (or other CT imaging system, such as an enclosed gantry helicalmulti-slice CT with a curved detector or tomotherapy system) andgenerate an update to the current structural estimate of the region ofinterest as a function of the first simulated X-ray measurement and thefirst real X-ray measurement. A real X-ray measurement, as referred toherein, is an actual output that is received from a CBCT system (orother CT imaging system, such as an enclosed gantry helical multi-sliceCT with a curved detector or tomotherapy system) that represents theamount of X-rays received by and detected by the X-ray detector alongdifferent directions, such as in an image form. The update can begenerated invariant on (independent of) the current structural estimate.The structural estimate can be used to control one or more radiotherapytreatment parameters by recalculating dose, adjusting one or moreradiotherapy treatment machine parameters, or generating a display ofthe structural estimate on a graphical user interface.

In some embodiments, the software programs 144 generate the firstsimulated X-ray measurement based on the current structural estimate ofthe region of interest by applying the current structural estimate to amodel that generates an expected output of a real X-ray measurement. Insome implementations, the model that generates an expected output of areal X-ray measurement includes a modeling function representing beamhardening from a polyenergetic source. In some implementations, themodeling function that generates an expected output of a real X-raymeasurement includes a machine learning technique that is trained toestablish a relationship between a training real X-ray measurement and atraining known simulated X-ray measurement. In some implementations, themodeling function that generates an expected output of a real X-raymeasurement includes a linear model or a non-linear model that fitsX-ray data to some nominal value comprising at least one of relativeelectron density, mass density, monoenergetic attenuation, protonstopping power, or bone mineral density.

As an example, the model generates the expected output of the real X-raymeasurement, for a given measurement index i per number of X-raydetector elements in the CBCT system, in accordance with Equation 1:

z _(i) =b _(i)*exp(−[Ax]_(i))+r _(i),  (1)

where b is an incident intensity of flood field, A is a systemprojection matrix describing a combination of each image pixel at eachdetector element, x is the current X-ray attenuation map, exp is anexponential function, and r is background noise including scatter. Forexample, the model applies the current structural estimate to Equation 1as x to output the first simulated measurement z. The first simulatedmeasurement can be generated for each of a plurality of X-ray detectorelements that are in the CBCT system.

In an example, the value for r representing the noise including scattercan be re-estimated based on the current structural estimate at eachiteration. For example, the noise including scatter for a giveniteration can be applied to another machine learning technique or modelincluding x (e.g., the current structural estimate) to generate anupdate and re-estimate the value of the noise including scatter.

In some embodiments, the software programs 144 compute the update to thecurrent structural estimate as a derivative of a statistical objectivefunction with respect to the first simulated X-ray measurement. Forexample, the statistical objective function can be computed according toEquation 2:

A ^(T)(y./(b.*exp(−Ax)+r)−1)  (2)

where A is a system projection matrix describing a combination of eachimage pixel at each detector element, A^(T) is a transpose of the systemprojection matrix, y is the first real X-ray measurement, b is anincident intensity of flood field, x is the current X-ray attenuationmap, exp is an exponential function, and r is background noise includingscatter. The update to the current structural estimate represents aperturbation to the current structural estimate. Namely, the update islinearly projected to the current structural estimate into an imagespace to form a perturbation. The perturbation is scaled by a scalar forstability and the scaled perturbation is subtracted from the currentstructural estimate to generate an updated structural estimate.

In some implementations, after the structural estimate is updated, atleast one of regularization, momentum or denoising is applied to theupdated structural estimate to improve the image quality of the 3D modelrepresented by the structural estimate.

In some embodiments, the computation of the update and the updating ofthe structural estimate is conditioned on and performed in response to acomputation of a negative log-likelihood (NLL) function (e.g., a lossfunction). Specifically, the loss function is computed as a combinationof the first simulated X-ray measurement and the first real X-raymeasurement, such as in accordance with Equation 3:

NLL=sum(z _(i) −y _(i)*log(z _(i)))  (3)

which may represent a sum of the differences between each of theplurality of simulated and real measurements of the X-ray CBCT system.Namely, a plurality of simulated measurements each associated with adifferent X-ray detector can be generated in accordance with Equation 1.The corresponding real X-ray measurements can be received from the X-raydetector, such as from the image acquisition device 132. The differencesbetween each simulated measurement and the corresponding realmeasurement can be computed and summed. If the sum of these differencesis below a threshold (e.g., satisfies a stopping criterion), then theupdate to the structural estimate is not performed. On the other hand,in response to determining that the loss fails to satisfy the stoppingcriterion (e.g., if the sum of these differences is above thethreshold), the update to the current structural estimate is performed.

In some cases, instead of or in addition to conditioning the performanceof the update on the sum of the differences or the loss function definedby Equation 3, the stopping criterion (e.g., the condition for notperforming the update) can include a difference between adjacent updatesfalling below a threshold. For example, the software programs 144 canaccess a previous update value(s) and can generate a new update valuebased on the currently generated simulated X-ray measurement. Thesoftware programs 144 can compute a difference between these twoadjacent or subsequent update values to the structural estimate. Thesoftware programs 144 can compare this difference to a stoppingthreshold and if the difference is below the stopping threshold, thesoftware programs 144 may avoid performing the update to the structuralestimate. If the difference is above the stopping threshold, thesoftware programs 144 can perform the update to the structural estimate,such as in accordance with Equation 2.

In some cases, instead of or in addition to conditioning the performanceof the update on the sum of the differences or the loss function definedby Equation 3, the stopping criterion (e.g., the condition for notperforming the update) can include a number of iterations falling belowa maximum iteration value and/or an elapsed time falling below a maximumtime limit. For example, software programs 144 can keep a running countof the total number of iterations performed in which at each iterationthe update to the structural estimate is performed, such as inaccordance with Equation 2. The software programs 144 can also start atimer when the first update is performed. If the total number ofiterations is less than a maximum iteration value, then the softwareprograms 144 can perform the update to the structural estimate; and ifotherwise, then the update is not performed. As another example, if thetotal time since the first update was performed is less than the maximumtime limit, then the software programs 144 can perform the update to thestructural estimate; and if otherwise, then the update is not performed.

In some cases, instead of or in addition to conditioning the performanceof the update on the sum of the differences or the loss function definedby Equation 3, the stopping criterion (e.g., the condition for notperforming the update) can include input requesting termination anddisplay of a result. For example, a user input can be received duringthe process of performing the iterative image reconstruction. The inputmay request display of the current structural estimate. In response, thesoftware programs 144 stop performing the update and display the currentstructural estimate. User input can be received to resume the process ofperforming the iterative image reconstruction. In response, the softwareprograms 144 perform the last generated update and continue performingadditional updates until another stopping criterion is met or satisfied.

For example, the software programs 144 access an updated structuralestimate of the region of interest. The software programs 144 generate asecond simulated X-ray measurement based on the updated structuralestimate of the region of interest, such as in accordance withEquation 1. The software programs 144 receive a second real X-raymeasurement from the CBCT system. The software programs 144 generate afurther update to the updated structural estimate of the region ofinterest as a function of the second simulated X-ray measurement and thesecond real X-ray measurement, such as in accordance with Equation 2.

In some examples, the software programs 144 reduce a number ofiterations and efficiency of generating the updates to the structuralestimate by portioning the X-ray measurements into measurement groupsand only performing an update if a given real X-ray measurement fallsinto a group associated with updating the structural estimate.Specifically, the software programs 144 receive a plurality of realX-ray measurements. The X-ray measurements are partitioned intomeasurement groups, in which a first measurement group of themeasurement groups corresponds to a group of X-ray measurements used toupdate the current X-ray attenuation map, and a second measurement groupof the measurement groups corresponds to a group of X-ray measurementsthat is skipped from updating the current X-ray structural estimate. Thesoftware programs 144 determines that the first real X-ray measurementfalls within the first measurement group and, in response, the softwareprograms 144 perform the update to the current structural estimate.

In some cases, the measurement groups are divided by number or quantityof real X-ray measurements that are received. For example, a queue maybe maintained in which the real X-ray measurements are input. The queuemay include a number of entries. As each new real X-ray measurement isreceived, the X-ray measurement is input to the queue. When the numberof entries in the queue reaches a specified threshold, the last receivedX-ray measurement or a subset of last received X-ray measurements areused to perform an update to the structural estimate, such as inaccordance with Equation 2. For example, when the number of entries inthe queue reaches the specified threshold, the software programs 144 useEquation 1 to generate a simulated set of X-ray measurements which areused in combination with the last received X-ray measurement or a subsetof last received X-ray measurements to generate the update to thecurrent structural estimate. After the current structural estimate isupdated, the queue is flushed and the entries are deleted so that 0entries remain in the queue. In this way, the next update is performedwhen the next set of real X-ray measurements fill up the entries in thequeue to reach the specified threshold.

In some implementations, the current structural estimate is smoothedbased on a collection of updates to the structural estimate.Specifically, the software programs 144 can add each computed newupdated of the structural estimate to a collection of updates performedon the structural estimate. Each update in the collection corresponds toor is associated with a respective iteration of performing updates tothe structural estimate. The software programs 144 can identify apattern in the collected plurality of updates. The software programs 144can adjust or perform a new update to the structural estimate based onthe identified pattern in the collected plurality of updates. Forexample, the software programs 144 can generate a plurality of updatesbased on a plurality of previously generated simulated X-raymeasurements and corresponding real X-ray measurements. Each of theseupdates is input to a collection and a machine learning model orheuristic is applied to the collection to identify a particular pattern.The software programs 144 can then generate a new simulated X-raymeasurement and receive a new real X-ray measurement. The softwareprograms 144 can compute a new update to the structural estimate basedon the new simulated X-ray measurement and new real X-ray measurement,according to Equation 2. The software programs 144 can adjust thecomputed new update based on the particular pattern that has beenidentified in the collection of previous updates. This can smooth theupdates by preventing performance of updates that are too large in aparticular portion of the structural estimate.

In addition to the memory device 116 storing the software programs 144,it is contemplated that software programs 144 may be stored on aremovable computer medium, such as a hard drive, a computer disk, aCD-ROM, a DVD, a HD, a Blu-Ray DVD. USB flash drive, a SD card, a memorystick, or any other suitable medium; and the software programs 144 whendownloaded to image processing device 112 may be executed by imageprocessor 114.

The processor 114 may be communicatively coupled to the memory device116, and the processor 114 may be configured to executecomputer-executable instructions stored thereon. The processor 114 maysend or receive medical images 146 to memory device 116. For example,the processor 114 may receive medical images 146 from the imageacquisition device 132 via the communication interface 118 and network120 to be stored in memory device 116. The processor 114 may also sendmedical images 146 stored in memory device 116 via the communicationinterface 118 to the network 120 be either stored in database 124 or thehospital database 126.

Further, the processor 114 may utilize software programs 144 (e.g., atreatment planning software) along with the medical images 146 andpatient data 145 to create the radiation therapy treatment plan 142.Medical images 146 may include information such as imaging dataassociated with a patient anatomical region, organ, or volume ofinterest segmentation data. Patient data 145 may include informationsuch as (1) functional organ modeling data (e.g., serial versus parallelorgans, appropriate dose response models, etc.); (2) radiation dosagedata (e.g., DVH information); or (3) other clinical information aboutthe patient and course of treatment (e.g., other surgeries,chemotherapy, previous radiotherapy, etc.).

In addition, the processor 114 may utilize software programs to generateintermediate data such as updated parameters to be used, for example, bya machine learning model, such as a neural network model; or generateintermediate 2D or 3D images, which may then subsequently be stored inmemory device 116. The processor 114 may subsequently transmit theexecutable radiation therapy treatment plan 142 via the communicationinterface 118 to the network 120 to the radiation therapy device 130,where the radiation therapy plan will be used to treat a patient withradiation. In addition, the processor 114 may execute software programs144 to implement functions such as image conversion, image segmentation,deep learning, neural networks, and artificial intelligence. Forinstance, the processor 114 may execute software programs 144 that trainor contour a medical image; such software programs 144 when executed maytrain a boundary detector or utilize a shape dictionary.

The processor 114 may be a processing device, including one or moregeneral-purpose processing devices such as a microprocessor, a centralprocessing unit (CPU), a graphics processing unit (GPU), an acceleratedprocessing unit (APU), or the like. More particularly, the processor 114may be a complex instruction set computing (CISC) microprocessor, areduced instruction set computing (RISC) microprocessor, a very longinstruction Word (VLIW) microprocessor, a processor implementing otherinstruction sets, or processors implementing a combination ofinstruction sets. The processor 114 may also be implemented by one ormore special-purpose processing devices such as an application-specificintegrated circuit (ASIC), a field programmable gate array (FPGA), adigital signal processor (DSP), a System on a Chip (SoC), or the like.As would be appreciated by those skilled in the art, in someembodiments, the processor 114 may be a special-purpose processor ratherthan a general-purpose processor. The processor 114 may include one ormore known processing devices, such as a microprocessor from thePentium™, Core™, Xeon™, or Itanium® family manufactured by Intel™, theTurion™, Athlon™, Sempron™, Opteron™, FX™, Phenom™ family manufacturedby AMD™, or any of various processors manufactured by Sun Microsystems.The processor 114 may also include graphical processing units such as aGPU from the GeForce®, Quadro®, Tesla® family manufactured by Nvidia™,GMA, Iris™ family manufactured by Intel™, or the Radeon™ familymanufactured by AMD™. The processor 114 may also include acceleratedprocessing units such as the Xeon Phi™ family manufactured by Intel™.The disclosed embodiments are not limited to any type of processor(s)otherwise configured to meet the computing demands of identifying,analyzing, maintaining, generating, and/or providing large amounts ofdata or manipulating such data to perform the methods disclosed herein.In addition, the term “processor” may include more than one processor(for example, a multi-core design or a plurality of processors eachhaving a multi-core design). The processor 114 can execute sequences ofcomputer program instructions, stored in memory device 116, to performvarious operations, processes, methods that will be explained in greaterdetail below.

The memory device 116 can store medical images 146. In some embodiments,the medical images 146 may include one or more MRI images (e.g., 2D MRI,3D MRI, 2D streaming MRI, four-dimensional (4D) MRI, 4D volumetric MRI,4D cine MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI,diffusion MRI), CT images (e.g., 2D CT, cone beam CT, 3D CT, 4D CT),ultrasound images (e.g., 2D ultrasound, 3) ultrasound, 4D ultrasound),one or more projection images representing views of an anatomy depictedin the MRI, synthetic CT (pseudo-CT), and/or CT images at differentangles of a gantry relative to a patient axis. PET images, X-ray images,fluoroscopic images, radiotherapy portal images. SPECT images, computergenerated synthetic images (e.g., pseudo-CT images), aperture images,graphical aperture image representations of MLC leaf positions atdifferent gantry angles, and the like. Further, the medical images 146may also include medical image data, for instance, training images,ground truth images, contoured images, and dose images. In anembodiment, the medical images 146 may be received from the imageacquisition device 132. Accordingly, image acquisition device 132 mayinclude an MRI imaging device, a CT imaging device, a PET imagingdevice, an ultrasound imaging device, a fluoroscopic device, a SPECTimaging device, an integrated linac and MRI imaging device, or othermedical imaging devices for obtaining the medical images of the patient.The medical images 146 may be received and stored in any type of data orany type of format that the image processing device 112 may use toperform operations consistent with the disclosed embodiments.

The memory device 116 may be a non-transitory computer-readable medium,such as a read-only memory (ROM), a phase-change random access memory(PRAM), a static random access memory (SRAM), a flash memory, a randomaccess memory (RAM), a dynamic random access memory (DRAM) such assynchronous DRAM (SDRAM), an electrically erasable programmableread-only memory (EEPROM), a static memory (e.g., flash memory, flashdisk, static random access memory) as well as other types of randomaccess memories, a cache, a register, a CD-ROM, a DVD or other opticalstorage, a cassette tape, other magnetic storage device, or any othernon-transitory medium that may be used to store information includingimage, data, or computer-executable instructions (e.g., stored in anyformat) capable of being accessed by the processor 114, or any othertype of computer device. The computer program instructions can beaccessed by the processor 114, read from the ROM, or any other suitablememory location, and loaded into the RAM for execution by the processor114. For example, the memory device 116 may store one or more softwareapplications. Software applications stored in the memory device 16 mayinclude, for example, an operating system 143 for common computersystems as well as for software-controlled devices. Further, the memorydevice 116 may store an entire software application, or only a part of asoftware application, that is executable by the processor 114. Forexample, the memory device 116 may store one or more radiation therapytreatment plans 142.

The image processing device 112 can communicate with the network 120 viathe communication interface 118, which can be communicatively coupled tothe processor 114 and the memory device 116. The communication interface118 may provide communication connections between the image processingdevice 112 and radiotherapy system 100 components (e.g., permitting theexchange of data with external devices). For instance, the communicationinterface 118 may, in some embodiments, have appropriate interfacingcircuitry to connect to the user interface 136, which may be a hardwarekeyboard, a keypad, or a touch screen through which a user may inputinformation into radiotherapy system 100.

Communication interface 118 may include, for example, a network adaptor,a cable connector, a serial connector, a USB connector, a parallelconnector, a high-speed data transmission adaptor (e.g., such as fiber.USB 3.0, thunderbolt, and the like), a wireless network adaptor (e.g.,such as a WiFi adaptor), a telecommunication adaptor (e.g., 3G, 4G/LTEand the like), and the like. Communication interface 118 may include oneor more digital and/or analog communication devices that permit imageprocessing device 112 to communicate with other machines and devices,such as remotely located components, via the network 120.

The network 120 may provide the functionality of a local area network(LAN), a wireless network, a cloud computing environment (e.g., softwareas a service, platform as a service, infrastructure as a service, etc.),a client-server, a wide area network (WAN), and the like. For example,network 120 may be a LAN or a WAN that may include other systems S1(138), S2(140), and S3 (141). Systems S1, S2, and S3 may be identical toimage processing device 112 or may be different systems. In someembodiments, one or more systems in network 120 may form a distributedcomputing/simulation environment that collaboratively performs theembodiments described herein. In some embodiments, one or more systemsS1. S2, and S3 may include a CT scanner that obtains CT images (e.g.,medical images 146). In addition, network 120 may be connected toInternet 122 to communicate with servers and clients that resideremotely on the internet.

Therefore, network 120 can allow data transmission between the imageprocessing device 112 and a number of various other systems and devices,such as the OIS 128, the radiation therapy device 130, and the imageacquisition device 132. Further, data generated by the OIS 128 and/orthe image acquisition device 132 may be stored in the memory device 116,the database 124, and/or the hospital database 126. The data may betransmitted/received via network 120, through communication interface118 in order to be accessed by the processor 114, as required.

The image processing device 112 may communicate with database 124through network 120 to send/receive a plurality of various types of datastored on database 124. For example, database 124 may include machinedata (control points) that includes information associated with aradiation therapy device 130, image acquisition device 132, or othermachines relevant to radiotherapy. Machine data information may includecontrol points, such as radiation beam size, arc placement, beam on andoff time duration, machine parameters, segments, MLC configuration,gantry speed. MRI pulse sequence, and the like. Database 124 may be astorage device and may be equipped with appropriate databaseadministration software programs. One skilled in the art wouldappreciate that database 124 may include a plurality of devices locatedeither in a central or a distributed manner.

In some embodiments, database 124 may include a processor-readablestorage medium (not shown) While the processor-readable storage mediumin an embodiment may be a single medium, the term “processor-readablestorage medium” should be taken to include a single medium or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers) that store the one or more sets ofcomputer-executable instructions or data. The term “processor-readablestorage medium” shall also be taken to include any medium that iscapable of storing or encoding a set of instructions for execution by aprocessor and that cause the processor to perform any one or more of themethodologies of the present disclosure. The term “processor-readablestorage medium” shall accordingly be taken to include, but not belimited to, solid-state memories and optical and magnetic media. Forexample, the processor-readable storage medium can be one or morevolatile, non-transitory, or non-volatile tangible computer-readablemedia.

Image processor 114 may communicate with database 124 to read imagesinto memory device 116 or store images from memory device 116 todatabase 124. For example, the database 124 may be configured to store aplurality of images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CTimages, 2D Fluoroscopy images, X-ray images, raw data from MR scans orCT scans. Digital Imaging and Communications in Medicine (DIMCOM) data,projection images, graphical aperture images, etc.) that the database124 received from image acquisition device 132. Database 124 may storedata to be used by the image processor 114 when executing softwareprogram 144 or when creating radiation therapy treatment plans 142.Database 124 may store the data produced by the trained machine learningmode, such as a neural network including the network parametersconstituting the model learned by the network and the resultingestimated data. As referred to herein, “estimate” or “estimated” can beused interchangeably with “predict” or “predicted” and should beunderstood to have the same meaning. The image processing device 112 mayreceive the imaging data, such as a medical image 146 (e.g., 2D MRIslice images. CT images, 2D Fluoroscopy images, X-ray images, 3D MRIimages, 4D MRI images, projection images, graphical aperture images,image contours, etc.) from the database 124, the radiation therapydevice 130 (e.g., an MR-linac), and/or the image acquisition device 132to generate a treatment plan 142.

In an embodiment, the radiotherapy system 100 can include an imageacquisition device 132 that can acquire medical images (e.g., MRIimages, 3D MRI, 2D streaming MRI, 4D volumetric MRI, CT images,cone-Beam CT, PET images, functional MRI images (e.g., fMRI, DCE-MRI anddiffusion MRI), X-ray images, fluoroscopic image, ultrasound images,radiotherapy portal images, SPECT images, and the like) of the patient.Image acquisition device 132 may, for example, be an MRI imaging device,a CT imaging device, a PET imaging device, an ultrasound device, afluoroscopic device, a SPECT imaging device, or any other suitablemedical imaging device for obtaining one or more medical images of thepatient. Images acquired by the image acquisition device 132 can bestored within database 124 as either imaging data and/or test data Byway of example, the images acquired by the image acquisition device 132can be also stored by the image processing device 112 as medical images146 in memory device 116.

In an embodiment, for example, the image acquisition device 132 may beintegrated with the radiation therapy device 130 as a single apparatus(e.g., an MR-linac). Such an MR-linac can be used, for example, todetermine a location of a target organ or a target tumor in the patient,so as to direct radiation therapy accurately according to the radiationtherapy treatment plan 142 to a predetermined target.

The image acquisition device 132 can be configured to acquire one ormore images of the patient's anatomy for a region of interest (e.g., atarget organ, a target tumor, or both). Each image, typically a 2D imageor slice, can include one or more parameters (e.g., a 2D slicethickness, an orientation, and a location, etc.). In an embodiment, theimage acquisition device 132 can acquire a 2D slice in any orientation.For example, an orientation of the 2D slice can include a sagittalorientation, a coronal orientation, or an axial orientation. Theprocessor 114 can adjust one or more parameters, such as the thicknessand/or orientation of the 2D slice, to include the target organ and/ortarget tumor. In an embodiment, 2D slices can be determined frominformation such as a 3D MRI volume. Such 2D slices can be acquired bythe image acquisition device 132 in “real-time” while a patient isundergoing radiation therapy treatment, for example, when using theradiation therapy device 130, with “real-time” meaning acquiring thedata in at least milliseconds or less.

The image processing device 112 may generate and store radiation therapytreatment plans 142 for one or more patients. The radiation therapytreatment plans 142 may provide information about a particular radiationdose to be applied to each patient. The radiation therapy treatmentplans 142 may also include other radiotherapy information, such ascontrol points including beam angles, gantry angles, beam intensity,dose-histogram-volume information, the number of radiation beams to beused during therapy, the dose per beam, and the like.

The image processor 114 may generate the radiation therapy treatmentplan 142 by using software programs 144 such as treatment planningsoftware (such as Monaco®, manufactured by Elekta AB of Stockholm,Sweden). In order to generate the radiation therapy treatment plans 142,the image processor 114 may communicate with the image acquisitiondevice 132 (e.g., a CT device, an MRI device, a PET device, an X-raydevice, an ultrasound device, etc.) to access images of the patient andto delineate a target, such as a tumor, to generate contours of theimages. In some embodiments, the delineation of one or more OARs, suchas healthy tissue surrounding the tumor or in close proximity to thetumor, may be required. Therefore, segmentation of the OAR may beperformed when the OAR is close to the target tumor. In addition, if thetarget tumor is close to the OAR (e.g., prostate in near proximity tothe bladder and rectum), then by segmenting the OAR from the tumor, theradiotherapy system 100 may study the dose distribution not only in thetarget but also in the OAR.

In order to delineate a target organ or a target tumor from the OAR,medical images, such as MRI images, CT images, PET images, fMRI images,X-ray images, ultrasound images, radiotherapy portal images. SPECTimages, and the like, of the patient undergoing radiotherapy may beobtained non-invasively by the image acquisition device 132 to revealthe internal structure of a body part. Based on the information from themedical images, a 3D structure of the relevant anatomical portion may beobtained and used to generate a contour of the image. Contours of theimage can include data overlaid on top of the image that delineates oneor more structures of the anatomy. In some cases, the contours can befiles associated with respective images that specify the coordinates or2D or 3D locations of various structures of the anatomy depicted in theimages.

In addition, during a treatment planning process, many parameters may betaken into consideration to achieve a balance between efficienttreatment of the target tumor (e.g., such that the target tumor receivesenough radiation dose for an effective therapy) and low irradiation ofthe OAR(s) (e.g., the OAR(s) receives as low a radiation dose aspossible). Other parameters that may be considered include the locationof the target organ and the target tumor, the location of the OAR, andthe movement of the target in relation to the OAR. For example, the 3Dstructure may be obtained by contouring the target or contouring the OARwithin each 2D layer or slice of an MRI or CT image and combining thecontour of each 2D layer or slice. The contour may be generated manually(e.g., by a physician, dosimetrist, or health care worker using aprogram such as MONACO™ manufactured by Elekta AB of Stockholm. Sweden)or automatically (e.g., using a program such as the Atlas-basedauto-segmentation software. ABAS™, manufactured by Elekta AB ofStockholm, Sweden). In certain embodiments, the 3D structure of a targettumor or an OAR may be generated automatically by the treatment planningsoftware.

After the target tumor and the OAR(s) have been located and delineated,a dosimetrist, physician, or healthcare worker may determine a dose ofradiation to be applied to the target tumor, as well as any maximumamounts of dose that may be received by the OAR proximate to the tumor(e.g., left and right parotid, optic nerves, eyes, lens, inner ears,spinal cord, brain stem, and the like). After the radiation dose isdetermined for each anatomical structure (e.g., target tumor, OAR), aprocess known as inverse planning may be performed to determine one ormore treatment plan parameters that would achieve the desired radiationdose distribution. Examples of treatment plan parameters include volumedelineation parameters (e.g., which define target volumes, contoursensitive structures, etc.), margins around the target tumor and OARs,beam angle selection, collimator settings, and beam-on times. During theinverse-planning process, the physician may define dose constraintparameters that set bounds on how much radiation an OAR may receive(e.g., defining full dose to the tumor target and zero dose to any OAR:defining 95% of dose to the target tumor, defining that the spinal cord,brain stem, and optic structures receive ≤45 Gy, ≤55 Gy and <54 Gy,respectively). The result of inverse planning may constitute a radiationtherapy treatment plan 142 that may be stored in memory device 116 ordatabase 124. Some of these treatment parameters may be correlated. Forexample, tuning one parameter (e.g., weights for different objectives,such as increasing the dose to the target tumor) in an attempt to changethe treatment plan may affect at least one other parameter, which inturn may result in the development of a different treatment plan. Thus,the image processing device 112 can generate a tailored radiationtherapy treatment plan 142 having these parameters in order for theradiation therapy device 130 to provide radiotherapy treatment to thepatient.

In addition, the radiotherapy system 100 may include a display device134 and a user interface 136. The display device 134 may include one ormore display screens that display medical images, interface information,treatment planning parameters (e.g., projection images, graphicalaperture images, contours, dosages, beam angles, etc.) treatment plans,a target, localizing a target and/or tracking a target, or any relatedinformation to the user. The user interface 136 may be a keyboard, akeypad, a touch screen or any type of device that a user may use toinput information to radiotherapy system 100. Alternatively, the displaydevice 134 and the user interface 136 may be integrated into a devicesuch as a tablet computer (e.g., Apple iPad®, Lenovo Thinkpad®, SamsungGalaxy®, etc.).

Furthermore, any and all components of the radiotherapy system 100 maybe implemented as a virtual machine (e.g., VMWare, Hyper-V. and thelike). For instance, a virtual machine can be software that functions ashardware. Therefore, a virtual machine can include at least one or morevirtual processors, one or more virtual memories, and one or morevirtual communication interfaces that together function as hardware. Forexample, the image processing device 112, the OIS 128, and the imageacquisition device 132 could be implemented as a virtual machine. Giventhe processing power, memory, and computational capability available,the entire radiotherapy system 100 could be implemented as a virtualmachine.

FIG. 2A illustrates an example radiation therapy device 202 that mayinclude a radiation source, such as an X-ray source or a linearaccelerator, a couch 216, an imaging detector 214, and a radiationtherapy output 204. The radiation therapy device 202 may be configuredto emit a radiation beam 208 to provide therapy to a patient. Theradiation therapy output 204 can include one or more attenuators orcollimators, such as an MLC as described in the illustrative embodimentof FIG. 5, below.

Referring back to FIG. 2A, a patient can be positioned in a region 212and supported by the treatment couch 216 to receive a radiation therapydose, according to a radiation therapy treatment plan. The radiationtherapy output 204 can be mounted or attached to a gantry 206 or othermechanical support. One or more chassis motors (not shown) may rotatethe gantry 206 and the radiation therapy output 204 around couch 216when the couch 216 is inserted into the treatment area. In anembodiment, gantry 206 may be continuously rotatable around couch 216when the couch 216 is inserted into the treatment area. In anotherembodiment, gantry 206 may rotate to a predetermined position when thecouch 216 is inserted into the treatment area. For example, the gantry206 can be configured to rotate the therapy output 204 around an axis(“A”). Both the couch 216 and the radiation therapy output 204 can beindependently moveable to other positions around the patient, such asmoveable in transverse direction (“T”), moveable in a lateral direction(“V”), or as rotation about one or more other axes, such as rotationabout a transverse axis (indicated as “R”). A controller communicativelyconnected to one or more actuators (not shown) may control the couch's216 movements or rotations in order to properly position the patient inor out of the radiation beam 208 according to a radiation therapytreatment plan. Both the couch 216 and the gantry 206 are independentlymoveable from one another in multiple degrees of freedom, which allowsthe patient to be positioned such that the radiation beam 208 canprecisely target the tumor. The MLC may be integrated and includedwithin gantry 206 to deliver the radiation beam 208 of a certain shape.

The coordinate system (including axes A. T, and L) shown in FIG. 2A canhave an origin located at an isocenter 210. The isocenter 210 can bedefined as a location where the central axis of the radiation beam 208intersects the origin of a coordinate axis, such as to deliver aprescribed radiation dose to a location on or within a patient.Alternatively, the isocenter 210 can be defined as a location where thecentral axis of the radiation beam 208 intersects the patient forvarious rotational positions of the radiation therapy output 204 aspositioned by the gantry 206 around the axis A. As discussed herein, thegantry angle corresponds to the position of gantry 206 relative to axisA, although any other axis or combination of axes can be referenced andused to determine the gantry angle.

Gantry 206 may also have an attached imaging detector 214. The imagingdetector 214 is preferably located opposite to the radiation source, andin an embodiment, the imaging detector 214 can be located within a fieldof the therapy beam 208.

The imaging detector 214 can be mounted on the gantry 206 (preferablyopposite the radiation therapy output 204), such as to maintainalignment with the therapy beam 208. The imaging detector 214 rotatesabout the rotational axis as the gantry 206 rotates. In an embodiment,the imaging detector 214 can be a flat panel detector (e.g., a directdetector or a scintillator detector). In this manner, the imagingdetector 214 can be used to monitor the therapy beam 208 or the imagingdetector 214 can be used for imaging the patient's anatomy, such asportal imaging (e.g., to provide real X-ray measurements). The controlcircuitry of radiation therapy device 202 may be integrated withinsystem 100 or remote from it.

In an illustrative embodiment, one or more of the couch 216, the therapyoutput 204, or the gantry 206 can be automatically positioned, and thetherapy output 204 can establish the therapy beam 208 according to aspecified dose for a particular therapy delivery instance. A sequence oftherapy deliveries can be specified according to a radiation therapytreatment plan, such as using one or more different orientations orlocations of the gantry 206, couch 216, or therapy output 204. Thetherapy deliveries can occur sequentially, but can intersect in adesired therapy locus on or within the patient, such as at the isocenter210. A prescribed cumulative dose of radiation therapy can thereby bedelivered to the therapy locus while damage to tissue near the therapylocus can be reduced or avoided.

FIG. 2B illustrates an example radiation therapy device 202 that mayinclude a combined linac and an imaging system, such as can include a CTimaging system. The radiation therapy device 202 can include an MLC (notshown). The CT imaging system can include an imaging X-ray source 218,such as providing X-ray energy in a kiloelectron-Volt (keV) energy rangewhich can be used for imaging the patient's anatomy, such as portalimaging (e.g., to provide real X-ray measurements). The imaging X-raysource 218 can provide a fan-shaped and/or a conical beam 208 directedto an imaging detector 222, such as a flat panel detector. The radiationtherapy device 202 can be similar to the system described in relation toFIG. 2A, such as including a radiation therapy output 204, a gantry 206,a couch 216, and another imaging detector 214 (such as a flat paneldetector). The X-ray source 218 can provide a comparatively-lower-energyX-ray diagnostic beam, for imaging.

In the illustrative embodiment of FIG. 2B, the radiation therapy output204 and the X-ray source 218 can be mounted on the same rotating gantry206, rotationally-separated from each other by 90 degrees. In anotherembodiment, two or more X-ray sources can be mounted along thecircumference of the gantry 206, such as each having its oil detectorarrangement to provide multiple angles of diagnostic imagingconcurrently. Similarly, multiple radiation therapy outputs 204 can beprovided.

FIG. 3 depicts an example radiation therapy system 300 that can includecombining a radiation therapy device 202 and an imaging system, such asa nuclear MR imaging system (e.g., known in the art as an MR-linac)consistent with the disclosed embodiments. As shown, system 300 mayinclude a couch 216, an image acquisition device 320, and a radiationdelivery device 330. System 300 delivers radiation therapy to a patientin accordance with a radiotherapy treatment plan. In some embodiments,image acquisition device 320 may correspond to image acquisition device132 in FIG. 1 that may acquire origin images of a first modality (e.g.,MRI image shown in FIG. 4A) or destination images of a second modality(e.g., CT image shown in FIG. 4B).

Couch 216 may support a patient (not shown) during a treatment session.In some implementations, couch 216 may move along a horizontaltranslation axis (labelled “I”), such that couch 216 can move thepatient resting on couch 216 into and/or out of system 300. Couch 216may also rotate around a central vertical axis of rotation, transverseto the translation axis. To allow such movement or rotation, couch 216may have motors (not shown) enabling the couch 216 to move in variousdirections and to rotate along various axes. A controller (not shown)may control these movements or rotations in order to properly positionthe patient according to a treatment plan.

In some embodiments, image acquisition device 320 may include an MRImachine used to acquire 2D or 3D MRI images of the patient before,during, and/or after a treatment session. Image acquisition device 320may include a magnet 321 for generating a primary magnetic field formagnetic resonance imaging. The magnetic field lines generated byoperation of magnet 321 may run substantially parallel to the centraltranslation axis I. Magnet 321 may include one or more coils with anaxis that runs parallel to the translation axis I. In some embodiments,the one or more coils in magnet 321 may be spaced such that a centralwindow 323 of magnet 321 is free of coils. In other embodiments, thecoils in magnet 321 may be thin enough or of a reduced density such thatthey are substantially transparent to radiation of the wavelengthgenerated by radiotherapy device 330. Image acquisition device 320 mayalso include one or more shielding coils, which may generate a magneticfield outside magnet 321 of approximately equal magnitude and oppositepolarity in order to cancel or reduce any magnetic field outside ofmagnet 321. As described below, radiation source 331 of radiotherapydevice 330 may be positioned in the region where the magnetic field iscancelled, at least to a first order, or reduced.

Image acquisition device 320 may also include two gradient coils 325 and326, which may generate a gradient magnetic field that is superposed onthe primary magnetic field. Coils 325 and 326 may generate a gradient inthe resultant magnetic field that allows spatial encoding of the protonsso that their position can be determined. Gradient coils 325 and 326 maybe positioned around a common central axis with the magnet 321 and maybe displaced along that central axis. The displacement may create a gap,or window, between coils 325 and 326. In embodiments where magnet 321can also include a central window 323 between coils, the two windows maybe aligned with each other.

In some embodiments, image acquisition device 320 may be an imagingdevice other than an MRI, such as an X-ray, a CT, a CBCT, a spiral CT, aPET, a SPECT, an optical tomography, a fluorescence imaging, ultrasoundimaging, radiotherapy portal imaging device, or the like. As would berecognized by one of ordinary skill in the art, the above description ofimage acquisition device 320 concerns certain embodiments and is notintended to be limiting.

Radiotherapy device 330 may include the radiation source 331, such as anX-ray source or a linac, and an MLC 332 (shown below in FIG. 5)Radiotherapy device 330 may be mounted on a chassis 335. One or morechassis motors (not shown) may rotate chassis 335 around couch 216 whencouch 216 is inserted into the treatment area. In an embodiment, chassis335 may be continuously rotatable around couch 216 when couch 216 isinserted into the treatment area. Chassis 335 may also have an attachedradiation detector (not shown), preferably located opposite to radiationsource 331 and with the rotational axis of chassis 335 positionedbetween radiation source 331 and the detector. Further, device 330 mayinclude control circuitry (not shown) used to control, for example, oneor more of couch 216, image acquisition device 320, and radiotherapydevice 330. The control circuitry of radiotherapy device 330 may beintegrated within system 300 or remote from it.

During a radiotherapy treatment session, a patient may be positioned oncouch 216. System 300 may then move couch 216 into the treatment areadefined by magnet 321, coils 325 and 326, and chassis 335. Controlcircuitry may then control radiation source 331, MLC 332, and thechassis motor(s) to deliver radiation to the patient through the windowbetween coils 325 and 326 according to a radiotherapy treatment plan.

FIG. 2A, FIG. 2B, and FIG. 3 illustrate generally embodiments of aradiation therapy device configured to provide radiotherapy treatment toa patient, including a configuration where a radiation therapy outputcan be rotated around a central axis (e.g., an axis “A”). Otherradiation therapy output configurations can be used. For example, aradiation therapy output can be mounted to a robotic arm or manipulatorhaving multiple degrees of freedom. In yet another embodiment, thetherapy output can be fixed, such as located in a region laterallyseparated from the patient, and a platform supporting the patient can beused to align a radiation therapy isocenter with a specified targetlocus within the patient.

As discussed above, radiation therapy devices described by FIG. 2A, FIG.2B, and FIG. 3 include an MLC for shaping, directing, or modulating anintensity of a radiation therapy beam to the specified target locuswithin the patient. FIG. 5 illustrates an example MLC 332 that includesleaves 532A through 532J that can be automatically positioned to definean aperture approximating a tumor 540 cross section or projection. Theleaves 532A through 532J permit modulation of the radiation therapybeam. The leaves 532A through 532J can be made of a material specifiedto attenuate or block the radiation beam in regions other than theaperture, in accordance with the radiation treatment plan. For example,the leaves 532A through 532J can include metallic plates, such ascomprising tungsten, with a long axis of the plates oriented parallel toa beam direction and having ends oriented orthogonally to the beamdirection (as shown in the plane of the illustration of FIG. 2A) A“state” of the MLC 332 can be adjusted adaptively during a course ofradiation therapy treatment, such as to establish a therapy beam thatbetter approximates a shape or location of the tumor 540 or anothertarget locus. This is in comparison to using a static collimatorconfiguration or as compared to using an MLC 332 configurationdetermined exclusively using an “offline” therapy planning technique. Aradiation therapy technique using the MLC 332 to produce a specifiedradiation dose distribution to a tumor or to specific areas within atumor can be referred to as IMRT

FIG. 6 illustrates an embodiment of another type of radiotherapy device630 (e.g., a Leksell Gamma Knife), according to some embodiments of thepresent disclosure. As shown in FIG. 6, in a radiotherapy treatmentsession, a patient 602 may wear a coordinate frame 620 to keep stablethe patient's body part (e.g., the head) undergoing surgery orradiotherapy. Coordinate frame 620 and a patient positioning system 622may establish a spatial coordinate system, which may be used whileimaging a patient or during radiation surgery. Radiotherapy device 630may include a protective housing 614 to enclose a plurality of radiationsources 612. Radiation sources 612 may generate a plurality of radiationbeams (e.g., beamlets) through beam channels 616. The plurality ofradiation beams may be configured to focus on an isocenter 210 fromdifferent directions. While each individual radiation beam may have arelatively low intensity, isocenter 210 may receive a relatively highlevel of radiation when multiple doses from different radiation beamsaccumulate at isocenter 210. In certain embodiments, isocenter 210 maycorrespond to a target under surgery or treatment, such as a tumor.

FIG. 7 illustrates an example flow diagram for deep learning, where adeep learning model (or a machine learning model), such as a deepconvolutional neural network (DCNN), can be trained and used todetermine a simulated X-ray measurement and/or an update to a structuralmodel based on a simulated X-ray measurement and a real X-raymeasurement. Namely, in one example, one or more DCNN models can betrained to establish a relationship between a training currentstructural model (e.g., an X-ray attenuation map) and a training knownsimulated X-ray measurement. In another example, one or more DCNN modelscan be trained to establish a relationship between a pair of trainingsimulated X-ray measurement and real X-ray measurement and a trainingknow % n update to a structural model (e.g., an X-ray attenuation map).

Inputs 704 can include a defined deep learning model (which can includeone or more sub-networks or one or more individual and independentmachine learning models) having an initial set of values and trainingdata. The training data can include patient images, structural models,simulated X-ray measurements, real X-ray measurements, and expectedresults. The training data can also include paired data sets, such as apair of a current structural model and a known simulated X-raymeasurement. The training data can also include paired data sets, suchas a pair of training simulated X-ray measurement and real X-raymeasurement and a training known update to a structural model. Thetraining data can include multiple of these paired data sets formultiple patients.

The deep learning model can include one or more neural networks(referred to as sub-networks), such as a DCNN. The deep learning networkcan be trained on the training data. For example, the deep learningnetwork can be trained based on multiple batches of paired structuralmodels and known simulated X-ray measurements. The deep learning networkcan be trained based on multiple batches of paired simulated X-raymeasurement and real X-ray measurement and known updates to a structuralmodel. In one embodiment, the deep learning network is trained in anend-to-end manner in which all of the sub-networks are trainedsimultaneously by being applied to a same set or batch of training dataand minimizing a set of cost functions. In another embodiment, one ormore of the sub-networks of the DCNN are trained separately andindependently in sequence by minimizing a set of cost functionsassociated with each particular sub-network.

The training real X-ray measurements can include images of an anatomy orregion of interest provided by one or more CT images, PET images, X-rayimages, or MRI images across one or more treatment fractions. Whentrained, the deep learning network can produce a simulated X-raymeasurement; and/or the deep learning network can produce an update to astructural model based on a simulated X-ray measurement and a real X-raymeasurement. In one implementation, the expected results can include theknown simulated X-ray measurement associated with a given structuralmodel. In another implementation, the expected results can include theknown update to the given structural model associated with a pair ofsimulated and real X-ray measurements.

During training of a first of the deep learning (DL) model 708, a batchof training data can be selected from the pairs of a training structuralmodel and expected results (e.g., the corresponding ground-truthsimulated X-ray measurement). In the case of end-to-end training, thebatch of training data can be processed by all of the sub-networks ofthe DL model 708 simultaneously. In this case, a set of cost functionsis minimized, the set of cost functions including a term based on adifference between an estimated simulated X-ray measurement produced bythe first DL model 708 and the ground-truth simulated X-ray measurementof the training structural model. The set of cost functions may also bea combination of individual cost functions that act on various networkoutputs. In the case of individual and sequential training of thesub-networks, the same or a different set of training data may be usedto train each sub-network.

The first deep learning model 708 can be applied to the selectedtraining structural model to provide estimated results (e.g., estimatedsimulated X-ray measurements), which can then be compared to theexpected results (e.g., ground truth simulated X-ray measurementsassociated with the training structural model) to compute a differenceor deviation that can provide an indication of training errors. Theerrors can be used during a procedure called backpropagation to updatethe parameters of the first deep learning network (e.g., layer nodeweights and biases of each or of certain sub-networks of the model 708),in order to reduce or minimize errors during subsequent trials. Theerrors can be compared to predetermined criteria, such as proceeding toa sustained minimum for a specified number of training iterations. Ifthe errors do not satisfy the predetermined criteria, then modelparameters of the first deep learning model 708 can be updated usingbackpropagation, and another batch of training data can be selected fromthe other sets of training data (of the same patient or other patients)and expected results for another iteration of deep learning modeltraining. If the errors satisfy the predetermined criteria, then thetraining can be ended, and the trained first model 708 can then be usedduring a deep learning testing or inference stage 712 to estimatesimulated X-ray measurements based on structural models received duringone or more treatment fractions. The trained first model 708 can receivea new current structural model and provide estimated results (e.g., thesimulated X-ray measurements for the current structural model).

After updating the parameters of the DCNN, the iteration index can beincremented by a value of one. The iteration index can correspond to anumber of times that the parameters of the DCNN have been updated.Stopping criteria can be computed, and if the stopping criteria aresatisfied, then the DCNN model can be saved in a memory, such as thememory device 116 of image processing device 112, and the training canbe halted. If the stopping criteria are not satisfied, then the trainingcan continue by obtaining another batch of training data from the sametraining subject or another training subject. In an embodiment, thestopping criteria can include a value of the iteration index (e.g., thestopping criteria can include whether the iteration index is greaterthan or equal to a determined maximum number of iterations). In anembodiment, the stopping criteria can include an accuracy of the outputsimulated X-ray measurement (e.g., the stopping criteria can includewhether the difference between the output simulated X-ray measurementand the ground-truth simulated X-ray measurement in the batch oftraining data is smaller than a threshold).

After the first DL model 708 is trained, a current structural model ofan anatomy can be received from an image acquisition device, such asimage acquisition device 132. A trained DCNN model can be received froma network, such as the network 120, or from a memory, such as the memorydevice 116 of image processing device 112. The trained DCNN can be usedto determine the estimated simulated X-ray measurement of the currentstructural model. This simulated X-ray measurement can be compared withthe real X-ray measurement to generate an update to the currentstructural model, as discussed above and below.

During training of a second of the deep learning (DL) model 708, a batchof training data can be selected from the pairs of a training simulatedX-ray measurement and real X-ray measurement and expected results (e.g.,the corresponding ground-truth update to a structural model). In thecase of end-to-end training, the batch of training data can be processedby all of the sub-networks of the DL model 708 simultaneously. In thiscase, a set of cost functions is minimized, the set of cost functionsincluding a term based on a difference between an estimated update tothe structural model produced by the second DL model 708 and theground-truth update to the structural model. The set of cost functionsmay also be a combination of individual cost functions that act onvarious network outputs. In the case of individual and sequentialtraining of the sub-networks, the same or different set of training datamay be used to train each sub-network.

The second deep learning model 708 can be applied to the selectedtraining simulated X-ray measurement and real X-ray measurement toprovide estimated results (e.g., estimated update to the structuralmodel), which can then be compared to the expected results (e.g., groundtruth update to the structural model) to compute a difference ordeviation that can provide an indication of training errors. The errorscan be used during a procedure called backpropagation to update theparameters of the second deep learning network (e.g., layer node weightsand biases of each or of certain sub-networks of the model 708), inorder to reduce or minimize errors during subsequent trials. The errorscan be compared to predetermined criteria, such as proceeding to asustained minimum for a specified number of training iterations. If theerrors do not satisfy the predetermined criteria, then model parametersof the second deep learning model 708 can be updated usingbackpropagation, and another batch of training data can be selected fromthe other sets of training data (of the same patient or other patients)and expected results for another iteration of deep learning modeltraining. If the errors satisfy the predetermined criteria, then thetraining can be ended, and the trained second model 708 can then be usedduring a deep learning testing or inference stage 712 to estimateupdates to a structural model based on simulated and real X-raymeasurements received during one or more treatment fractions. Thetrained second model 708 can receive a new set of simulated and realX-ray measurements and provide estimated results (e.g., the estimatedupdate to the current structural model based on the set of simulated andreal X-ray measurements).

After updating the parameters of the DCNN, the iteration index can beincremented by a value of one. The iteration index can correspond to anumber of times that the parameters of the DCNN have been updated.Stopping criteria can be computed, and if the stopping criteria aresatisfied, then the DCNN model can be saved in a memory, such as thememory device 116 of image processing device 112, and the training canbe halted. If the stopping criteria are not satisfied, then the trainingcan continue by obtaining another batch of training data from the sametraining subject or another training subject. In an embodiment, thestopping criteria can include a value of the iteration index (e.g., thestopping criteria can include whether the iteration index is greaterthan or equal to a determined maximum number of iterations). In anembodiment, the stopping criteria can include an accuracy of the outputestimated update to the structural model (e.g., the stopping criteriacan include whether the difference between the estimated update to thestructural model and the ground-truth update to the structural model inthe batch of training data is smaller than a threshold).

After the second DL model 708 is trained, a current set of simulated andreal X-ray measurements associated with an anatomy can be received froman image acquisition device, such as image acquisition device 132. Insome cases, the simulated X-ray measurement can be received by thesecond DL model 708 from the first DL model 708 based on the currentstructural model. A trained DCNN model can be received from a network,such as the network 120, or from a memory, such as the memory device 116of image processing device 112. The trained DCNN can be used todetermine the estimated update to the current structural model based onthe set of simulated and real X-ray measurements.

Specifically, in one example, a current structural model is accessed.The first DL model 708 is applied to the current structural model toestimate a simulated X-ray measurement based on the current structuralmodel. The simulated X-ray measurement generated by the first DL model708 and a real X-ray measurement can be applied to a second DL model708. The second DL model 708 estimates an update to the currentstructural model based on the simulated X-ray measurement generated bythe first DL model 708 and the real X-ray measurement that is received.The update generated by the second DL model 708 can then be used tomodify or make changes to the current structural model.

FIG. 8 illustrates an example data flow of a process 800 for trainingand use of a machine learning model to generate a simulated X-raymeasurement or update to a structural estimate, according to someembodiments of the present disclosure. The process 800 may be embodiedin computer-readable instructions for execution by one or moreprocessors such that the operations of the process 800 may be performedin part or in whole by the functional components of the image processingdevice 112; accordingly, the process 800 is described below by way ofexample with reference thereto. However, in other embodiments, at leastsome of the operations of the process 800 may be deployed on variousother hardware configurations. The process 800 is therefore not intendedto be limited to the image processing device 12 and can be implementedin whole, or in part, by any other component. Some or all of theoperations of process 800 can be in parallel, out of order, or entirelyomitted.

At operation 810, image processing device 112 receives training data.For example, image processing device 112 receives training data, whichmay include paired training data sets (e.g., input-output trainingpairs).

At operation 820, image processing device 112 receives one or more costfunctions for training the model.

At operation 830, image processing device 112 performs training of themodel based on the received training data and one or more costfunctions.

At operation 850, image processing device 112 outputs the trained model.For example, image processing device 112 outputs the trained model tooperate on a new set of X-ray measurements or a new structural estimate.

At operation 860, image processing device 112 utilizes the trained modelto generate a simulated X-ray measurement or update to a structuralestimate.

FIG. 9 illustrates a process 900 for generating an update to astructural estimate of a region of interest, according to someembodiments of the present disclosure. The process 900 may be embodiedin computer-readable instructions for execution by one or moreprocessors such that the operations of the process 900 may be performedin part or in whole by the functional components of the image processingdevice 112, accordingly, the process 900 is described below by way ofexample with reference thereto. However, in other embodiments, at leastsome of the operations of the process 900 may be deployed on variousother hardware configurations. The process 900 is therefore not intendedto be limited to the image processing device 112 and can be implementedin whole, or in part, by any other component. Some or all of theoperations of process 900 can be in parallel, out of order, or entirelyomitted.

At operation 910, image processing device 112 accesses a currentstructural estimate of a region of interest, as discussed above.

At operation 920, image processing device 112 generates a firstsimulated X-ray measurement based on the current structural estimate ofthe region of interest, as discussed above.

At operation 930, image processing device 112 receives a first realX-ray measurement from a CBCT system, as discussed above.

At operation 940, image processing device 112 generates an update to thecurrent structural estimate of the region of interest as a function ofthe first simulated X-ray measurement and the first real X-raymeasurement, the update being generated invariant on the currentstructural estimate, as discussed above.

FIG. 10 depicts the differences between structural estimates generatedaccording to different techniques, according to some embodiments of thepresent disclosure. Specifically, the images shown in FIG. 10 areiterative reconstructions of an anthropomorphic head phantom scanned ona cone-beam CT system. For example, a typical iteration reconstructionof an image is shown in image 1010. Image 1010 shows the result ofiteration reconstruction (window [−985,1000] HU) of the anthropomorphichead phantom scanned on a cone-beam CT system based on minimizing aweighted-least squares (WLS) objective function after 50 iterations.Image 1020 shows the result of another typical iteration reconstructionbased on minimizing the Poisson negative log-likelihood through monotoneseparable quadratic surrogates (SQS) for 150 iterations. Image 1030shows a result of performing iterative reconstruction in accordance withthe disclosed techniques after 50 iterations. As shown, the image 1030resulting from the disclosed techniques includes less noise and phantomsthan the images 1010 and 1020 resulting from the typical approaches.This demonstrates the speed and quality improvements of the disclosedapproach over existing methods.

FIG. 11 illustrates a block diagram of an embodiment of a machine 1100on which one or more of the methods as discussed herein can beimplemented. In one or more embodiments, one or more items of the imageprocessing device 112 can be implemented by the machine 1100. Inalternative embodiments, the machine 1100 operates as a standalonedevice or may be connected (e.g., networked) to other machines. In oneor more embodiments, the image processing device 112 can include one ormore of the items of the machine 1100. In a networked deployment, themachine 1100 may operate in the capacity of a server or a client machinein server-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine 1100 maybe a personal computer (PC), a tablet PC, a set-top box (STB), aPersonal Digital Assistant (PDA), a cellular telephone, a web appliance,a network router, switch or bridge, or any machine capable of executinginstructions (sequential or otherwise) that specify actions to be takenby that machine. Further, while only a single machine 1100 isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein.

The example machine 1100 includes processing circuitry (e.g., theprocessor 1102, a CPU, a GPU, an ASIC, circuitry, such as one or moretransistors, resistors, capacitors, inductors, diodes, logic gates,multiplexers, buffers, modulators, demodulators, radios (e.g., transmitor receive radios or transceivers), sensors 1121 (e.g., a transducerthat converts one form of energy (e.g., light, heat, electrical,mechanical, or other energy) to another form of energy), or the like, ora combination thereof), a main memory 1104 and a static memory 1106,which communicate with each other via a bus 1108. The machine 1100(e.g., computer system) may further include a video display unit 1110(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). Themachine 1100 also includes an alphanumeric input device 1112 (e.g., akeyboard), a user interface (UI) navigation device 1114 (e.g., a mouse),a disk drive or mass storage unit 1116, a signal generation device 1118(e.g., a speaker), and a network interface device 1120.

The disk drive or mass storage unit 1116 includes a machine-readablemedium 1122 on which is stored one or more sets of data structures andinstructions (e.g., software) 1124 embodying or utilized by any one ormore of the methodologies or functions described herein. Theinstructions 1124 may also reside, completely or at least partially,within the main memory 1104 and/or within the processor 1102 duringexecution thereof by the machine 1100, the main memory 1104 and theprocessor 1102 also constituting machine-readable media.

The machine 1100 as illustrated includes an output controller 1128. Theoutput controller 1128 manages data flow to/from the machine 1100. Theoutput controller 1128 is sometimes called a device controller, withsoftware that directly interacts with the output controller 1128 beingcalled a device driver.

While the machine-readable medium 1122 is shown in an embodiment to be asingle medium, the term “machine-readable medium” may include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or moreinstructions 1124 or data structures. The term “machine-readable medium”shall also be taken to include any tangible medium that is capable ofstoring, encoding or carrying instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present disclosure, or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude non-volatile memory, including by way of example semiconductormemory devices, e.g., Erasable Programmable Read-Only Memory (EPROM),EEPROM, and flash memory devices; magnetic disks such as internal harddisks and removable disks; magneto-optical disks: and CD-ROM and DVD-ROMdisks.

The instructions 1124 may further be transmitted or received over acommunications network 1126 using a transmission medium. Theinstructions 1124 may be transmitted using the network interface device1120 and any one of a number of well-known transfer protocols (e.g.,HTTP). Examples of communication networks include a LAN, a WAN, theInternet, mobile telephone networks, Plain Old Telephone (POTS)networks, and wireless data networks (e.g., WiFi and WiMax networks).The term “transmission medium” shall be taken to include any intangiblemedium that is capable of storing, encoding or carrying instructions forexecution by the machine, and includes digital or analog communicationssignals or other intangible media to facilitate communication of suchsoftware.

As used herein. “communicatively coupled between” means that theentities on either of the coupling must communicate through an itemtherebetween and that those entities cannot communicate with each otherwithout communicating through the item.

ADDITIONAL NOTES

The above detailed description includes references to the accompanyingdrawings, which form a part of the detailed description. The drawingsshow, by way of illustration but not by way of limitation, specificembodiments in which the disclosure can be practiced. These embodimentsare also referred to herein as “examples.” Such examples can includeelements in addition to those shown or described. However, the presentinventors also contemplate examples in which only those elements shownor described are provided. Moreover, the present inventors alsocontemplate examples using any combination or permutation of thoseelements shown or described (or one or more aspects thereof), eitherwith respect to a particular example (or one or more aspects thereof),or with respect to other examples (or one or more aspects thereof) shownor described herein.

All publications, patents, and patent documents referred to in thisdocument are incorporated by reference herein in their entirety, asthough individually incorporated by reference. In the event ofinconsistent usages between this document and those documents soincorporated by reference, the usage in the incorporated reference(s)should be considered supplementary to that of this document; forirreconcilable inconsistencies, the usage in this document controls.

In this document, the terms “a,” “an,” “the,” and “said” are used whenintroducing elements of aspects of the disclosure or in the embodimentsthereof, as is common in patent documents, to include one or more thanone or more of the elements, independent of any other instances orusages of “at least one” or “one or more.” In this document, the term“or” is used to refer to a nonexclusive or, such that “A or B” includes“A but not B,” “B but not A,” and “A and B,” unless otherwise indicated.

In the appended claims, the terms “including” and “in which” are used asthe plain-English equivalents of the respective terms “comprising” and“wherein.” Also, in the following claims, the terms “comprising,”“including,” and “having” are intended to be open-ended to mean thatthere may be additional elements other than the listed elements, suchthat elements after such a term (e.g., comprising, including, having) ina claim are still deemed to fall within the scope of that claim.Moreover, in the following claims, the terms “first,” “second,” and“third,” and so forth, are used merely as labels, and are not intendedto impose numerical requirements on their objects.

Embodiments of the disclosure may be implemented withcomputer-executable instructions. The computer-executable instructions(e.g., software code) may be organized into one or morecomputer-executable components or modules. Aspects of the disclosure maybe implemented with any number and organization of such components ormodules. For example, aspects of the disclosure are not limited to thespecific computer-executable instructions or the specific components ormodules illustrated in the figures and described herein. Otherembodiments of the disclosure may include different computer-executableinstructions or components having more or less functionality thanillustrated and described herein.

Method examples (e.g., operations and functions) described herein can bemachine or computer-implemented at least in part (e.g., implemented assoftware code or instructions). Some examples can include acomputer-readable medium or machine-readable medium encoded withinstructions operable to configure an electronic device to performmethods as described in the above examples. An implementation of suchmethods can include software code, such as microcode, assembly languagecode, a higher-level language code, or the like (e.g., “source code”).Such software code can include computer-readable instructions forperforming various methods (e.g., “object” or “executable code”). Thesoftware code may form portions of computer program products. Softwareimplementations of the embodiments described herein may be provided viaan article of manufacture with the code or instructions stored thereon,or via a method of operating a communication interface to send data viaa communication interface (e.g., wirelessly, over the internet, viasatellite communications, and the like).

Further, the software code may be tangibly stored on one or morevolatile or non-volatile computer-readable storage media duringexecution or at other times. These computer-readable storage media mayinclude any mechanism that stores information in a form accessible by amachine (e.g., computing device, electronic system, and the like), suchas, but are not limited to, floppy disks, hard disks, removable magneticdisks, any form of magnetic disk storage media, CD-ROMS,magnetic-optical disks, removable optical disks (e.g., compact disks anddigital video disks), flash memory devices, magnetic cassettes, memorycards or sticks (e.g., secure digital cards), RAMs (e.g., CMOS RAM andthe like), recordable/non-recordable media (e.g., ROMs), EPROMS,EEPROMS, or any type of media suitable for storing electronicinstructions, and the like. Such computer-readable storage mediumcoupled to a computer system bus may be accessible by the processor andother parts of the OIS.

In an embodiment, the computer-readable storage medium may have encodeda data structure for a treatment planning, wherein the treatment planmay be adaptive. The data structure for the computer-readable storagemedium may be at least one of a Digital Imaging and Communications inMedicine (DICOM) format, an extended DICOM format, an XML format, andthe like. DICOM is an international communications standard that definesthe format used to transfer medical image-related data between varioustypes of medical equipment. DICOM RT refers to the communicationstandards that are specific to radiation therapy.

In various embodiments of the disclosure, the method of creating acomponent or module can be implemented in software, hardware, or acombination thereof. The methods provided by various embodiments of thepresent disclosure, for example, can be implemented in software by usingstandard programming languages such as, for example, Compute UnifiedDevice Architecture (CUDA), C, C++, Java, Python, and the like; andusing standard machine learning/deep learning library (or API), such astensorflow, torch and the like; and combinations thereof. As usedherein, the terms “software” and “firmware” are interchangeable, andinclude any computer program stored in memory for execution by acomputer.

A communication interface includes any mechanism that interfaces to anyof a hardwired, wireless, optical, and the like, medium to communicateto another device, such as a memory bus interface, a processor businterface, an Internet connection, a disk controller, and the like. Thecommunication interface can be configured by providing configurationparameters and/or sending signals to prepare the communication interfaceto provide a data signal describing the software content. Thecommunication interface can be accessed via one or more commands orsignals sent to the communication interface.

The present disclosure also relates to a system for performing theoperations herein. This system may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. The order of execution or performance of the operations inembodiments of the disclosure illustrated and described herein is notessential, unless otherwise specified. That is, the operations may beperformed in any order, unless otherwise specified, and embodiments ofthe disclosure may include additional or fewer operations than thosedisclosed herein. For example, it is contemplated that executing orperforming a particular operation before, contemporaneously with, orafter another operation is within the scope of aspects of thedisclosure.

In view of the above, it will be seen that the several objects of thedisclosure are achieved, and other beneficial results attained. Havingdescribed aspects of the disclosure in detail, it will be apparent thatmodifications and variations are possible without departing from thescope of aspects of the disclosure as defined in the appended claims. Asvarious changes could be made in the above constructions, products, andmethods without departing from the scope of aspects of the disclosure,it is intended that all matter contained in the above description andshown in the accompanying drawings shall be interpreted as illustrativeand not in a limiting sense.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Inaddition, many modifications may be made to adapt a particular situationor material to the teachings of the disclosure without departing fromits scope. While the dimensions, types of materials and coatingsdescribed herein are intended to define the parameters of thedisclosure, they are by no means limiting and are example embodiments.Many other embodiments will be apparent to those of skill in the artupon reviewing the above description. The scope of the disclosureshould, therefore, be determined with reference to the appended claims,along with the full scope of equivalents to which such claims areentitled.

Also, in the above Detailed Description, various features may be groupedtogether to streamline the disclosure. This should not be interpreted asintending that an unclaimed disclosed feature is essential to any claim.Rather, inventive subject matter may lie in less than all features of aparticular disclosed embodiment. Thus, the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment. The scope of the disclosure should bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled. Further, thelimitations of the following claims are not written inmeans-plus-function format and are not intended to be interpreted basedon 35 U.S.C. § 112, sixth paragraph, unless and until such claimlimitations expressly use the phrase “means for” followed by a statementof function void of further structure.

The Abstract is provided to comply with 37 C.F.R. § 1.72(b), to allowthe reader to quickly ascertain the nature of the technical disclosure.It is submitted with the understanding that it will not be used tointerpret or limit the scope or meaning of the claims.

What is claimed is:
 1. A system comprising: a memory; and one or moreprocessors that, when executing instructions stored in the memory, areconfigured to perform operations comprising: accessing a currentstructural estimate of a region of interest; generating a firstsimulated X-ray measurement based on the current structural estimate ofthe region of interest; receiving a first real X-ray measurement; andgenerating an update to the current structural estimate of the region ofinterest as a function of the first simulated X-ray measurement and thefirst real X-ray measurement, the update being generated invariant onthe current structural estimate.
 2. The system of claim 1, wherein thefirst real X-ray measurement is received from a cone-beam computedtomography (CBCT) system or computed tomography (CT) system, and whereinthe operations further comprise: computing the update as a derivative ofa statistical objective function with respect to the first simulatedX-ray measurement.
 3. The system of claim 1, wherein the operationsfurther comprise: linearly projecting the update to the currentstructural estimate into an image space to form a perturbation; scalingthe perturbation by a scalar for stability; and subtracting the scaledperturbation from the current structural estimate to generate an updatedstructural estimate.
 4. The system of claim 3, wherein the operationsfurther comprise: applying at least one of regularization, momentum ordenoising to the updated structural estimate.
 5. The system of claim 1,wherein the current structural estimate comprises an X-ray attenuationmap that represents a three-dimensional (3D) model of the region ofinterest.
 6. The system of claim 5, wherein the X-ray attenuation mapcomprises a linear X-ray attenuation map.
 7. The system of claim 1,wherein generating the first simulated X-ray measurement based on thecurrent structural estimate of the region of interest comprises applyingthe current structural estimate to a model that generates an expectedoutput of a real X-ray measurement.
 8. The system of claim 7, whereinthe current structural estimate comprises an X-ray attenuation map, andwherein the model generates the expected output of the real X-raymeasurement, for a given measurement index i per number of detectorelements in a cone-beam computed tomography (CBCT) system, in accordancewith:z _(i) =b _(i)*exp(−[Ax]_(i))+r _(i), where b is an incident intensityof flood field, A is a system projection matrix describing a combinationof each image pixel at each detector element, x is the current X-rayattenuation map, exp is an exponential function, and r is backgroundnoise including scatter.
 9. The system of claim 8, wherein theoperations further comprise re-estimating r representing the backgroundnoise including scatter based on the current structural estimate at eachiteration of generating the update.
 10. The system of claim 8, whereinthe model comprises a modeling function representing beam hardening froma polyenergetic source, the modeling function comprising a machinelearning technique that is trained to establish a relationship between atraining real X-ray measurement and a training known simulated X-raymeasurement; a linear model; or a non-linear model that fits X-ray datato some nominal value comprising at least one of relative electrondensity, mass density, monoenergetic attenuation, proton stopping power,or bone mineral density.
 11. The system of claim 1, wherein theoperations further comprise repeating the generating of a simulatedX-ray measurement, receiving of a real X-ray measurement, and generatingof an update to the current structural estimate for multiple sets ofsimulated and real X-ray measurements.
 12. The system of claim 1,wherein the operations further comprise: accessing an updated structuralestimate of the region of interest; generating a second simulated X-raymeasurement based on the updated structural estimate of the region ofinterest; receiving a second real X-ray measurement; and generating afurther update to the updated structural estimate of the region ofinterest as a function of the second simulated X-ray measurement and thesecond real X-ray measurement.
 13. The system of claim 1, wherein theoperations further comprise: accessing an objective function comprisinga negative log-likelihood (NLL) function; computing a loss by applyingthe NLL function to a combination of the first simulated X-raymeasurement and the first real X-ray measurement; and in response todetermining that the loss fails to satisfy a criterion, performing theupdate to the current structural estimate, the criterion comprising adifference between adjacent updates falling below a threshold, a numberof iterations falling below a maximum iteration value, an elapsed timefalling below a maximum time limit, or input requesting termination anddisplay of a result.
 14. The system of claim 1, wherein the currentstructural estimate comprises an X-ray attenuation map, and wherein theupdate to the current structural estimate of the region of interest iscomputed in accordance with:A ^(T)(y/(b*exp(−Ax)+r)−1), where A is a system projection matrixdescribing a combination of each image pixel at each detector element,A^(T) is a transpose of the system projection matrix, y is the firstreal X-ray measurement, b is an incident intensity of flood field, x isthe current X-ray attenuation map, exp is an exponential function, and ris background noise including scatter.
 15. The system of claim 1,wherein the operations further comprise: receiving a plurality of realX-ray measurements; partitioning the plurality of real X-raymeasurements into measurement groups, a first measurement group of themeasurement groups corresponding to a group of X-ray measurements usedto update the current structural estimate, and a second measurementgroup of the measurement groups corresponding to a group of X-raymeasurements that is skipped from updating the current structuralestimate; determining that the first real-X-ray measurement falls withinthe first measurement group; and in response to determining that thefirst real-X-ray measurement falls within the first measurement group,performing the update to the current structural estimate.
 16. The systemof claim 1, wherein the operations further comprise: collecting aplurality of updates to the structural estimate, each of the pluralityof updates being associated with a respective iteration; identifying apattern of updates based on the collected plurality of updates; andperforming the update to the current structural estimate based on theidentified pattern of updates.
 17. A method comprising: accessing acurrent structural estimate of a region of interest; generating a firstsimulated X-ray measurement based on the current structural estimate ofthe region of interest; receiving a first real X-ray measurement; andgenerating an update to the current structural estimate of the region ofinterest as a function of the first simulated X-ray measurement and thefirst real X-ray measurement, the update being generated invariant onthe current structural estimate.
 18. The method of claim 17, furthercomprising: computing the update as a derivative of a statisticalobjective function with respect to the first simulated X-raymeasurement.
 19. The method of claim 17, further comprising: linearlyprojecting the update to the current structural estimate into an imagespace to form a perturbation; scaling the perturbation by a scalar forstability; and subtracting the scaled perturbation from the currentstructural estimate to generate an updated structural estimate.
 20. Themethod of claim 19, further comprising: applying at least one ofregularization, momentum or denoising to the updated structuralestimate.
 21. The method of claim 17, wherein the current structuralestimate comprises an X-ray attenuation map that represents athree-dimensional (3D) model of the region of interest.
 22. The methodof claim 21, wherein the X-ray attenuation map comprises a linear X-rayattenuation map.
 23. The method of claim 17, wherein generating thefirst simulated X-ray measurement based on the current structuralestimate of the region of interest comprises applying the currentstructural estimate to a model that generates an expected output of areal X-ray measurement.
 24. A non-transitory computer-readable mediumcomprising non-transitory computer-readable instructions that, whenexecuted by one or more processors, configure the one or more processorsto perform operations comprising: accessing a current structuralestimate of a region of interest; generating a first simulated X-raymeasurement based on the current structural estimate of the region ofinterest; receiving a first real X-ray measurement; and generating anupdate to the current structural estimate of the region of interest as afunction of the first simulated X-ray measurement and the first realX-ray measurement, the update being generated invariant on the currentstructural estimate.
 25. The non-transitory computer-readable medium ofclaim 24, wherein the operations further comprise: computing the updateas a derivative of a statistical objective function with respect to thefirst simulated X-ray measurement.
 26. The non-transitorycomputer-readable medium of claim 24, wherein the operations furthercomprise: linearly projecting the update to the current structuralestimate into an image space to form a perturbation; scaling theperturbation by a scalar for stability; and subtracting the scaledperturbation from the current structural estimate to generate an updatedstructural estimate.
 27. The non-transitory computer-readable medium ofclaim 26, wherein the operations further comprise: applying at least oneof regularization, momentum or denoising to the updated structuralestimate.
 28. The non-transitory computer-readable medium of claim 24,wherein the current structural estimate comprises an X-ray attenuationmap that represents a three-dimensional (3D) model of the region ofinterest.
 29. The non-transitory computer-readable medium of claim 28,wherein the X-ray attenuation map comprises a linear X-ray attenuationmap.
 30. The non-transitory computer-readable medium of claim 24,wherein generating the first simulated X-ray measurement based on thecurrent structural estimate of the region of interest comprises applyingthe current structural estimate to a model that generates an expectedoutput of a real X-ray measurement.